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Executive  Summary 


One  of  the  defining  properties  of  current  and  future  air  and  missile  defense  (AMD)  command 
and  control  (C2)  systems  is  an  increasing  reliance  on  automation.  Technological  opportunities 
and  an  increasingly  complex  operating  environment  have  created  a  situation  where  AMD 
operators  must  be  provided  with  automated  decision  support  to  meet  mission  objectives.  There 
is  a  tendency  among  system  developers  with  little  background  in  human  performance  issues  to 
assume  that  automation  is  innately  beneficial.  Research  in  a  number  of  areas  suggests,  however, 
that  such  is  not  always  the  case.  To  begin,  automation  elevates  operators  into  system  monitors 
rather  than  active  controllers.  Operators  are  thus  removed  from  moment-to-moment,  active 
control  and  become  monitors  and  managers  of  subordinate  automated  processes.  It  is  a  well- 
established  fact  that  humans  make  very  poor  system  monitors.  Beyond  classical  vigilance, 
research  and  operational  experience  indicate  that  automation  does  not  replace  human  operator 
tasks.  Rather,  it  changes  the  nature  of  the  work  that  operators  do,  and  it  does  this  in  ways  that 
are  often  unanticipated  by  system  developers  and  users.  Moreover,  the  preponderance  of  theory 
and  empirical  evidence  suggests  that  the  job  of  supervisory  controller  is  quite  different  from  that 
of  a  traditional  operator.  In  a  similar  vein,  it  has  been  shown  that  automation  does  not  reduce 
training  requirements;  it  changes  training  requirements  and  often  makes  job  preparation  more 
demanding. 

Other  human  performance  problems  associated  with  automation  generally  fall  into  one  of  two 
categories:  (1)  loss  of  situational  awareness  (SA)  and  (2)  skill  impairment.  SA  is  important 
because  it  has  been  shown  to  be  a  key  determinant  of  decision  quality  in  battle  command. 
Automation  in  and  of  itself  does  not  prevent  operators  from  establishing  or  maintaining  SA  or 
contribute  to  skill  impairment.  However,  improper  implementation  coupled  with  inadequate  or 
inappropriate  training  can  make  it  more  difficult  for  operators  to  establish  and  maintain  SA  and 
contribute  to  skill  development  and  retention  problems. 

Much  of  contemporary  automation  applied  to  real-time  C2  brings  into  play  what  has  been  called 
the  Catch-22  of  human  supervisory  control:  Automation  has  been  introduced  because  it  is 
thought  to  be  able  to  do  the  job  better  than  human  controllers,  but  humans  have  been  left  in  the 
control  loop  to  “monitor”  that  the  automated  system  is  performing  correctly  and  to  override  the 
automation  when  it  is  “wrong.”  The  unstated  assumption  is  that  human  operators  can  properly 
decide  when  the  automation’s  decisions  should  be  overridden.  Humans  are  expected  to 
compensate  for  machine  unreliability,  but  they  suffer  from  a  variety  of  cognitive  limitations  and 
vulnerabilities  that  make  it  nearly  impossible  to  meet  this  expectation.  A  number  of  automation 
researchers  have  thus  concluded  that  while  the  risks  associated  with  automation  unreliability  can 
never  be  eliminated  entirely,  they  can  be  managed  more  effectively  through  a  number  of  positive 
actions  directed  at  supporting  and  enhancing  effective  human  supervisory  control  (HSC).  These 
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actions  generally  fall  into  one  of  two  categories:  (1)  design  to  support  effective  HSC,  and  (2) 
training  to  support  effective  HSC.  This  report  addresses  training  to  support  effective  HSC. 

Section  1  begins  with  a  brief  review  of  results  from  the  Patriot  Vigilance  project.  This  effort  was 
concerned  with  the  human  perfonnance  contributors  to  fratricides  involving  the  Patriot  air 
defense  missile  system  during  Operation  Iraqi  Freedom.  Recommendations  from  the  Patriot 
Vigilance  project  indicated  that  the  AMD  community  must  address  two  primary  problems 
associated  with  automation  as  applied  in  current  Patriot  operations:  (1)  developing  effective 
HSC,  and  (2)  the  level  of  operator  expertise  required  to  employ  a  highly  automated  system  such 
as  Patriot  on  the  modem  battlefield.  These  issues  are  also  relevant  to  AMD  systems  under 
development  and  to  other  Army  systems  being  fielded  to  support  network-centric  warfare 
concepts. 

Section  2  opens  with  a  review  of  basic  steps  in  the  development  of  a  good  instructional  program. 
In  present  context,  a  good  instructional  program  is  one  that  supports  relevant  practice  with 
feedback  leading  to  desired  warfighting  competencies.  This  section  also  includes  discussion  of 
relevant  background  research  on  the  development  of  human  expertise.  The  section  concludes 
with  a  discussion  of  Training  Demons,  the  enemies  of  good  training.  Training  Demons  are 
training  outcomes  and  practices  that  occur  when  the  rules  of  proper  instructional  design  are  not 
followed.  These  enemies  of  good  training  include:  (1)  inadequate  job  and  task  analysis,  (2) 
misplaced  training  planning  emphases,  (3)  lock-step  versus  criterion-referenced  instruction,  (4) 
content-based  versus  skill-  and  job-oriented  training,  (5)  a  misplaced  emphasis  on  free-play 
versus  deliberate  practice,  (6)  inadequate  or  poorly-defined  perfonnance  standards,  (7)  lack  of 
emphasis  on  performance  feedback,  and  (8)  various  forms  of  training  folk  wisdom  that 
negatively  impact  training  delivery. 

Section  3  addresses  six  training  dilemmas  or  issues  that  remain  unresolved  and  stand  in  the  way 
of  moving  AMD  training  away  from  its  current  focus  on  crew-drills  in  the  direction  of  regarding 
operators  as  managers  of  lethal  and  complex  systems.  These  issues  include:  (1)  initial  skill 
development,  (2)  time  to  train  and  job  progression  patterns,  (3)  training  for  unreliable 
automation,  (4)  team  training,  (5)  training  for  adaptive  expertise,  and  (6)  mindfulness — 
developing  crews  and  C2  teams  that  are  capable  of  sustained  high  reliability  in  a  complex  and 
uncertain  operational  setting. 

Finally,  section  4  outlines  a  path  forward  for  AMD.  This  section  is  developed  in  two  parts:  (1) 
training  for  routine  expertise  and  (2)  training  for  adaptive  expertise — coping  with  automation 
unreliability  and  the  inevitable  fog  of  war.  The  subsection  on  training  for  routine  expertise 
addresses  six  focus  areas  for  improvement  in  AMD  training  for  routine  operations — when  things 
work  as  they  should.  These  focus  areas  are:  (1)  job  and  task  analysis,  (2)  scenario  content  and 
training  delivery,  (3)  performance  standards,  (4)  time  to  reach  desired  competency  levels,  (5) 
qualified  instructors,  and  (6)  performance  feedback — the  after  action  review  (AAR)  process. 
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Training  for  adaptive  expertise  is  intended  to  foster  the  ability  to  think  “outside  the  box”  defined 
by  routine  crew  drills.  Adaptive  expertise  is  built  on  a  solid  foundation  of  the  basics  of  AMD 
operations — training  for  routine  expertise.  Section  4  addresses  the  basic  requirements  for  the 
development  of  adaptive  individuals,  crews,  and  teams.  The  discussion  ends  negatively  by 
noting  that  three  major  roadblocks  stand  in  the  way  of  developing  adaptive  expertise:  (1) 
training  time  and  job  progression  practices,  (2)  training  quality  (aspects  of  the  AMD  branch’s 
traditional  training  practices),  and  (3)  trainee  motivation  to  develop  as  AMD  professionals. 
Unless  these  roadblocks  are  confronted  and  remedied,  adaptive  crews  and  C2  teams  are  likely  to 
be  the  exception  rather  than  the  rule.  As  the  Defense  Science  Board  has  cautioned,  the  risk  in 
leaving  things  mostly  as  they  are  is  that  training  failures  can  negate  promising  technology  and 
hardware. 


Intentionally  Left  Blank 


1.  Background 


1.1  Overview 

One  of  the  defining  properties  of  the  next  generation  of  air  and  missile  defense  (AMD)  command 
and  control  (C2)  systems  is  an  increasing  reliance  on  automation.  This  report  is  the  third  in  a 
series  of  three  dealing  with  human  perfonnance  and  training  issues  in  the  development  and 
effective  use  of  automated  AMD  C2  systems.  The  first  report  (Hawley,  Mares,  &  Giammanco, 
2005)  discusses  the  impact  of  automation  on  air  defense  operators  and  the  consequences  of  their 
role  change  from  traditional  operators  to  supervisory  controllers.  The  second  report  (Hawley  & 
Mares,  2006)  expands  upon  that  background  material  and  addresses  the  issue  of  developing 
effective  human  supervisory  control  (HSC)  in  AMD  C2  systems.  The  focus  of  this  report  is 
training  for  AMD  operators  and  the  battle  staff.  In  the  words  of  the  Army  Board  of  Inquiry 
(BOI)  investigating  the  Patriot  fratricides  that  occurred  during  Operation  Iraqi  Freedom  (OIF),  it 
is  necessary  to  re-look  the  level  of  expertise  required  to  operate  such  lethal  systems  on  the 
modem  battlefield.  Together,  these  reports  are  intended  as  a  primer  on  automation,  supervisory 
control,  and  effective  human  performance  for  commanders,  concept  developers,  system 
designers,  trainers,  and  other  personnel  involved  with  decision-making  and  operations  for  the 
next  generation  of  AMD  C2  systems. 

1.2  Concepts  and  Terms 

Prior  to  introducing  the  major  topics  in  the  report,  it  is  necessary  to  define  and  clarify  several 
terms  that  are  used  in  the  text.  These  definitions  and  clarifications  are  given  in  the  paragraphs  to 
follow. 

Sheridan  (1992,  p.  3)  defines  automation  as  “the  automatically  controlled  operation  of  an 
apparatus,  a  process,  or  a  system,  by  mechanical  or  electrical  devices  that  take  the  place  of 
human  organs  of  observation,  decision,  or  effort.”  In  contemporary  AMD  C2,  the  combination 
of  operational  complexity  and  technical  advances  have  led  to  a  situation  in  which  functions — 
perception,  decision-making,  response  selection  and  implementation — assigned  to  the  human 
subsystem  in  previous  generations  of  AMD  systems  are  now  assigned  to  the  machine  subsystem. 

Supervisory  control  is  defined  as  a  situation  in  which  “one  or  more  operators  are  continually 
programming  and  receiving  information  from  a  computer  that  interconnects  through  artificial 
effectors  and  sensors  to  the  controlled  process  or  task  environment”  (Sheridan,  2002,  p.  115). 
Under  a  supervisory  control  regimen,  operators  do  not  interact  with  the  controlled  process 
directly,  as  they  previously  did  in  manual  or  less  automated  systems.  Rather,  the  operators 
receive  information  from  and  provide  input  to  a  computer,  which,  in  turn,  directs  the  controlled 
process.  The  operator’s  role  is  thus  changed  from  direct,  on-line  process  control  to  supervisor  of 
a  mostly  computer-directed  process.  Their  job  is  to  supervise  the  computer  controller.  The 
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consequences  of  this  role  transformation — though  often  subtle — must  be  reflected  in  system 
design,  performance  support  feature  (i.e.,  job  aiding),  and  operator-controller1  training  and 
professional  development. 

Effective  HSC  means  that  AMD  operator-controllers  are  able  to  carry  out  their  explicit  role, 
which  is  to  supervise  or  direct  subordinate  automated  control  systems.  Effective  HSC  implies 
meaningful  human  oversight,  where  meaningful  human  oversight  means  that  operator-controller 
supervision  of  subordinate  automated  control  systems  is  actual  and  not  theoretical  or  abstract.  A 
third  related  term,  positive  human  control,  is  a  specific  aspect  of  effective  HSC  and  meaningful 
human  oversight.  Positive  human  control  means  that  firing  decisions — to  shoot  or  not  to  shoot — 
are  based  on  conscious  problem  solving  and  discernment  and  not  merely  the  result  of  automation 
bias,  or  uncritical  acquiescence  following  the  system’s  recommendation. 

Knowledge  refers  to  background  information  required  in  the  performance  of  a  job  or  task. 
Knowledge  is  sometimes  described  as  “knowing  about.”  Skill  refers  to  a  task  or  group  of  tasks 
performed  to  a  specific  level  of  proficiency,  which  often  use  motor  functions  and  typically 
require  the  manipulation  of  instruments  or  equipment.  Some  skills  are  knowledge-  and  attitude- 
based  and  do  not  require  equipment  manipulation.  Skills  are  sometimes  characterized  as 
“knowing  how  to  do.”  A  competency  is  a  set  of  skills  performed  to  a  specific  standard  under 
specific  conditions.  Competencies  are  sometimes  characterized  in  terms  of  the  level  of  knowing 
how  to  do.  Competencies  differ  from  task  standards  in  that  they  are  whole-job-referenced  rather 
than  individual  task  referenced. 

1.3  The  Patriot  Vigilance  Project 

Personnel  from  the  Anny  Research  Laboratory’s  Human  Research  and  Engineering  Directorate 
(HRED)  began  looking  into  Patriot  and  AMD  performance  and  training  issues  at  the  invitation  of 
the  then  Ft.  Bliss  Commander,  MG  Michael  A.  Vane.  MG  Vane  was  interested  in  operator 
vigilance  and  situational  awareness  (SA)  as  they  relate  to  the  performance  of  automated  AMD 
battle  command  systems.  The  generally  accepted  definition  of  SA  is  from  Endsley  (1996)  who 
defines  it  as  the  perception  of  elements  in  the  environment,  the  comprehension  of  their  meaning, 
and  the  projection  of  their  status  in  the  near  future.  MG  Vane  was  particularly  concerned  by 
what  he  termed  a  “lack  of  vigilance”  on  the  part  of  Patriot  operators  along  with  an  apparent  “lack 
of  cognizance”  of  what  was  being  presented  to  them  on  situation  displays  with  an  ensuing 
“absolute  trust  in  automation.”  His  request  for  human  engineering  support  was  prompted  by  an 
unacceptable  number  of  fratricide  incidents  by  Patriot  units  during  OIF. 


To  differentiate  traditional  manual  operators  from  operators  performing  in  a  HSC  setting,  these  latter  personnel  are  referred 
to  as  “operator-controllers.” 
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The  project  staff  spent  most  of  the  summer  and  fall  of  2004  perfonning  a  root  cause  analysis  of 
the  OIF  fratricide  incidents — reading  documents,  interviewing  knowledgeable  personnel  in  the 
Ft.  Bliss  area,  and  observing  Patriot  training  and  operations.  An  initial  report  was  delivered  to 
MG  Vane  in  October  2004.  A  discussion  of  the  results  of  what  was  tenned  the  Patriot  Vigilance 
project  is  presented  in  Hawley  and  Mares  (2006)  and  is  not  repeated  here,  other  than  in  a 
summary  manner.  HRED’s  report  to  MG  Vane  in  October  2004  recommended  two  primary 
actionable  items  to  redress  the  performance  problems  identified  during  the  Patriot  Vigilance 
effort: 

1.  Re-define  the  operators’  roles  to  provide  meaningful  human  oversight  of  system 
operations,  and 

2.  Develop  more  effective  Engagement  Operations  (EO)  personnel,  particularly  the  air  battle 
operations  (ABO)  slice  of  EO — re-look  the  level  of  expertise  required  to  operate  such  a 
lethal  system  on  the  modem  battlefield. 

The  Defense  Science  Board  (DSB)  (DSB,  2004)  reinforced  HRED’s  recommendations  with  the 
following  comments.  Although  the  full  DSB  report  on  Patriot  system  performance  is  classified, 
these  extracts  are  not. 

The  Patriot  system  should  migrate  to  more  of  a  “man-in-the-loop”  philosophy  versus  a 
fully  automated  philosophy — providing  operator  awareness  and  control  of  engagement 
processes. 

and 


Patriot  training  and  simulations  should  be  upgraded  to  support  this  man-in-the-loop 
protocol  including  the  ability  to  train  on  confusing  and  complex  scenarios  that  contain 
unbriefed  surprises. 

The  key  notion  in  the  first  DSB  recommendation  is  captured  in  the  phrase,  “providing  operator 
awareness  and  control  of  engagement  processes.”  Simply  put,  Soldiers  and  not  the  automated 
system  must  be  the  ultimate  decision  makers  in  AMD  engagements.  Decisions  to  shoot  or  not  to 
shoot  must  be  made  by  crews  having  adequate  SA  and  the  expertise  to  understand  the 
significance  of  the  information  available  to  them.  The  DSB’s  first  recommendation  is 
synonymous  with  HRED’s  first  actionable  item  concerning  establishing  meaningful  human 
oversight  of  Patriot  and  other  AMD  systems. 

The  second  DSB  recommendation  having  major  significance  for  human  performance  in 
contemporary  AMD  operations  concerns  operator-controller  training  and  professional 
development.  Here,  the  DSB  supported  HRED’s  conclusion  that  it  is  necessary  to  re-look  the 
level  of  expertise  necessary  to  operate  such  a  lethal  system  on  the  modem  battlefield.  To 
highlight  the  importance  of  the  training  issue,  the  fratricide  BOI  stated  bluntly  “The  system 
[Patriot]  is  too  lethal  to  be  placed  in  the  hands  of  crews  trained  to  such  a  limited  standard.” 
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The  BOI  examining  the  OIF  fratricides  specifically  criticized  Patriot  training  for  emphasizing 
“rote  drills”  versus  the  “exercise  of  high-level  judgment.”  The  essence  of  this  criticism  is  that 
the  user  community  approaches  training  for  a  complex,  knowledge-based  function  like  ABO  in 
much  the  same  manner  as  linear,  skill-  and  rule-based  actions  like  March  Order  and 
Emplacement  or  System  Set-up.  The  emphasis  is  on  mastering  routines  rather  than  adaptive 
problem  solving.  However,  the  range  of  actions  required  in  routine  drills  is  narrower  and  more 
predictable  than  those  encountered  in  combat  operations. 

The  U.S.  Navy  faced  a  similar  reconsideration  of  training  practices  in  the  aftermath  of  the  shoot- 
down  of  the  Iranian  airbus  by  the  USS  Vincennes  in  1988.  After  more  than  10  years  of  research, 
the  Navy  reached  several  conclusions  that  are  also  relevant  to  the  contemporary  AMD  setting. 
First,  the  Navy’s  research  concluded  that  Aegis  operator-controller  training  must  emphasize  the 
development  of  adaptive  decision-making  skills.  Adaptive  decision-making  skills,  or  the  ability 
to  think  outside  the  box  defined  by  routine  crew  drills,  are  a  key  aspect  of  effective  operator- 
controller  performance  in  ambiguous  situations.  The  second  major  conclusion  was  that 
shipboard  (i.e.,  unit)  training  must  address  team  in  addition  to  individual  perfonnance. 

Competent  crews  are  the  basis  of  effective  unit  performance,  and  crews  are  more  than  the  sum  of 
their  individual  members.  Both  of  these  issues  are  addressed  explicitly  in  the  sections  to  follow. 

The  DSB’s  recommendation  to  include  unbriefed  surprises  in  training  does  not  mean  that  it  is 
sufficient  merely  to  insert  anomalous  events  like  those  encountered  in  OIF  into  training 
scenarios.  In  advanced  AMD  training,  the  scenario  is  the  curriculum.  To  properly  prepare 
operator-controllers  for  combat,  scenario  designers  must  bear  in  mind  that  the  surprises  of  OIF 
are  representative  of  a  class  of  potential  anomalies.  Selected  anomalies  occurred  then;  others — 
some  similar,  some  different — will  occur  on  future  battlefields.  It  is  thus  necessary  that 
operator-controllers  be  imbued  with  a  sense  of  mindfulness  that  automated  battle  command 
systems  are  fallible.  The  system’s  recommendations  will  be  correct  most  but  not  all  of  the  time. 
Training  must  foster  the  development  of  the  expertise  essential  to  recognize  potential  anomalies 
and  the  skills  necessary  to  determine  an  appropriate  course  of  action.  Operator-controllers  must 
walk  a  fine  line  between  blind  faith  and  wholesale  mistrust.  AMD  decision  makers  must  not 
underestimate  the  difficulties  associated  with  adequately  meeting  this  training  challenge. 

1.4  The  Rest  of  the  Report 

The  idea  for  this  report  grew  out  of  a  conversation  with  the  Anny  Training  and  Doctrine 
Command  (TRADOC)  System  Manager-Lower  Tier  (TSM-LT),  during  which  he  suggested 
follow-on  reports  addressing  two  topics  beyond  those  addressed  initially  in  Hawley,  Mares,  and 
Giammanco  (2005): 

•  Design  for  effective  HSC  of  AMD  systems 

•  Training  for  effective  HSC  in  AMD  operations 
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The  present  report  addresses  the  second  follow-on  topic:  Training  for  effective  HSC.  In  the  next 
section,  we  begin  this  treatment  with  a  discussion  of  core  training  issues  in  AMD.  A  section 
addressing  various  unresolved  training  issues  for  research  and  experimentation  follows  this 
opening  discussion.  Finally,  we  discuss  a  path  forward  for  AMD.  The  path  forward  addresses 
elements  of  training  to  develop  both  routine  (that  required  for  normal  operations)  and  adaptive 
expertise  (that  required  for  situations  that  are  not  typically  encountered  during  routine 
operations).  In  keeping  with  the  focus  on  effective  HSC,  the  discussion  to  follow  emphasizes 
training  for  AMD  ABO — activities  performed  by  Patriot  crewmembers  in  the  battery-level 
Engagement  Control  Station  (ECS)  and  at  the  battalion-level  Infonnation  and  Coordination 
Central  (ICC). 


2.  Core  Training  Issues:  Revisiting  the  Basics 


2.1  Training  Preliminaries 

Training  is  a  process  by  which  job-relevant  knowledge,  skills,  and  competencies  are  acquired  by 
individuals,  crews,  and  multi-echelon  units.  In  a  military  setting,  its  purpose  is  to  deliver  warfare 
competence  where  and  when  it  is  needed  (DSB,  2003).  The  DSB  report  further  notes  that 
training  consists  of  relevant  practice  with  feedback.  In  essence,  therefore,  training  is  a  process 
by  which  learning  at  various  levels  (individual,  crew,  or  unit)  is  facilitated. 

Training  is  not  simulations,  networks,  virtual  environments,  or  games  even  if  they  are  labeled  as 
training  devices.  All  of  these  mechanisms  can  be  used  to  craft  an  environment  in  which  learning 
might  take  place,  but  they  are  not  training  per  se.  This  is  an  important  distinction:  Various 
methods  and  technologies  such  as  those  mentioned  above  can  be  used  to  support  training  if  they 
provide  an  environment  in  which  job-relevant  practice  with  appropriate  feedback  can  occur. 
Training  developers  must,  however,  first  know  what  must  be  practiced  and  the  appropriate  form 
of  essential  feedback. 

A  good  instructional  program  is  defined  as  one  that  supports  relevant  practice  with  appropriate 
feedback  leading  to  desired  warfare  competencies.  Developing  a  good  instructional  program  is  a 
relatively  simple  process.  However,  the  devil  can  be  in  the  details  of  how  the  various  steps  in  the 
process  are  carried  out.  The  steps  in  proper  instructional  design  are  listed  as  follows  (Whitmore, 
2002): 

1 .  Specify  the  job  performances  we  want  trainees  to  learn  to  do. 

2.  Break  those  job  performances  down  successively  into  their  component  activities  (including 
mental  activities)  until  we  reach  the  level  of  skill  already  possessed  by  the  trainees. 
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3.  Develop  situations  that  lead  trainees  through  those  things  we  want  them  to  learn  to  do. 

4.  Lead  trainees  through  practicing  the  new  activities,  beginning  with  the  lowest  level  and 
successively  assembling  higher-level  activities  from  lower-level  components. 

5.  Provide  trainees  with  sufficient  practice — with  feedback — at  each  level  to  reach  desired 
competency  levels. 

Steps  1  and  2  comprise  what  is  often  called  a  Job  and  Task  Analysis  (JTA),  or  Front-End 
Analysis  (FEA).  Step  3  is  an  aspect  of  course  design,  specifically  scenario  design.  Steps  4  and  5 
describe  proper  instructional  delivery.  Taken  together  and  without  all  of  the  detail,  this  process 
is  conceptually  similar  to  what  is  called  for  in  the  Instructional  Systems  Development  (ISD) 
process  or  its  Army  variant,  the  Systems  Approach  to  Training  (SAT). 

Before  leaving  the  present  topic,  we  want  to  comment  on  the  importance  of  the  JTA.  The 
information  obtained  from  a  JTA  must  be  based  on  a  system’s  contemporary  operating 
environment  and  not  simply  reflect  a  subjective  update  of  job  and  task  analysis  material  from  a 
previous  era.  Training  developers  are  often  tempted  to  pursue  this  latter  course  of  action  because 
comprehensive  job  and  task  analysis  can  be  an  expensive  and  time-consuming  undertaking.  A 
thorough  job  and  task  analysis  must  be  a  high  priority  for  current  systems  being  used  in  mission 
settings  different  from  that  originally  intended  (e.g.,  Patriot  against  a  tactical  ballistic  missile 
[TBM]  threat)  and  for  all  follow-on  systems.  Competent  JTA  involves  much  more  than  simply 
updating  an  existing  task  list  along  with  supporting  documentation.  In  their  2003  report,  the 
DSB  remarked  that  a  recurring  problem  with  Army  training  is  that  a  JTA  is  rarely  performed 
properly,  and  this  “original  sin”  returns  to  haunt  Army  training  over  the  life  cycle  of  most 
systems. 

It  has  been  noted  that  for  a  dynamic,  perfonnance-based  setting  like  AMD  ABO,  the  scenario  is 
the  curriculum  (Cannon-Bowers,  Burns,  Salas,  &  Pruitt,  1998).  The  relevant  practice  and 
feedback  essential  to  developing  warfare  competence  takes  place  within  a  simulation-based 
learning  environment.  In  this  setting,  training  scenarios  provide  the  structure  within  which  the 
stimulus  and  response  aspects  of  learning  are  presented  and  evaluated.  The  process  by  which 
effective  scenario-based  training  is  developed  and  implemented  is  depicted  in  figure  1 . 


6 


From  figure  1,  results  from  the  JTA  are  used  to  prepare  a  skill  inventory,  or  list  of  job  skills  to  be 
trained.  These  job-relevant  skills  are  used  to  develop  a  list  of  learning  objectives  and  associated 
competencies.  Learning  objectives  drive  the  development  of  scenario  events,  or  scripts.  If  you 
want  a  trainee  to  learn  how  to  do  something,  then  the  training  scenario  must  provide  the  stimuli 
necessary  to  elicit  that  behavior.  Next,  the  training  developer  must  define  perfonnance  measures 
and  standards.  A  performance  measure  is  a  behavioral  description  that  permits  an  evaluator  to 
unambiguously  decide  whether  a  performance  took  place.  The  performance  standard  provides  a 
way  of  judging  whether  that  perfonnance  was  acceptable. 

In  performance-oriented  training,  feedback  is  just  as  important  as  performance  evaluation.  For 
learning  to  occur,  trainees  must  be  provided  with  feedback  regarding  their  performance  along 
with  corrective  guidance.  Accordingly,  the  next  step  in  the  Scenario-Based  Training  Cycle  is 
performance  diagnosis:  If  the  performance  was  not  acceptable,  why  not?  Performance 
diagnostic  information  provides  the  basis  for  the  feedback  and  debrief  stage,  or  After  Action 
Review  (AAR).  Perfonnance  diagnosis  must  be  objective  and  specific  to  the  performances 
involved.  Moreover,  for  best  effect,  feedback  should  be  immediate  with  an  opportunity  to 
practice  less-than-successful  performance  elements. 

2.2  The  Development  of  Expertise 

Throughout  this  series  of  reports  we  have  referred  repeatedly  to  operator-controller  expertise.  In 
present  usage,  the  term  expertise  refers  to  a  capability  for  consistently  superior  performance  on  a 
specified  set  of  representative  tasks  for  a  domain  (Ericsson  &  Lehmann,  1996).  Expertise  in 
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AMD  ABO  is  derived  from  all  aspects  of  operator-controller  job  preparation:  (1)  traditional 
training  (institutional  and  unit),  (2)  professional  development  (self-directed  study  and 
professional  military  education),  and  (3)  relevant  on-the-job  experience. 

We  have  also  argued  that  SA  is  the  key  factor  determining  decision  quality  in  battle  command 
(Hawley,  Mares,  &  Giammanco,  2005).  SA  is  built  upon  in-depth  technical  and  tactical 
expertise.  The  primary  implication  of  this  conclusion  is  that  marginally-skilled  or  apprentice 
operator-controllers  cannot  develop  the  SA  necessary  for  effective  supervisory  control, 
regardless  of  the  sophistication  of  the  battle  command  hardware  suite  provided  to  them. 
Technology  can  support  and  amplify  human  expertise,  but  cannot  substitute  for  it. 

Given  the  centrality  of  operator-controller  expertise  to  effective  AMD  C2,  an  obvious  follow-on 
question  is,  “How  is  such  expertise  developed?”  Norman  (1993)  notes  that  there  are  at  least 
three  phases  of  learning  leading  to  expertise  as  defined  above.  These  are  (1)  accretion, 

(2)  tuning,  and  (3)  restructuring.  Accretion  is  the  accumulation  of  facts  (knowledge  in  our 
tenninology).  When  the  learner  has  a  proper  conceptual  framework  or  mental  model,  accretion 
is  easy,  painless,  and  efficient.  Without  a  good  conceptual  framework,  accretion  is  slow,  tedious, 
and  error-prone. 

Tuning  refers  to  the  process  of  translating  knowledge  into  skill.  This  requires  a  hands-on 
learning  environment  and  hours  and  hours  of  practice  under  the  supervision  of  a  coach  or 
mentor.  How  many  hours  are  necessary?  Norman  (1993)  asserts  that  for  any  complex  activity,  a 
minimum  of  5,000  hours  of  practice  (two  years  of  full-time  effort)  are  required  to  turn  a  novice 
into  an  entry-level  expert.  Other  researchers  such  as  Ericsson  and  Chamess  (1994)  put  the  time 
estimate  for  transitioning  from  novice  to  expert  at  10,000  hours  of  practice.  It  all  depends  on 
one’s  definition  of  an  expert. 

The  final  stage  of  learning  is  restructuring,  or  forming  and  reforming  the  proper  conceptual 
structure  for  performing  as  an  expert.  Norman  (1993)  remarks  that  accretion  and  tuning  are 
primarily  experiential — they  take  place  actively  in  an  experience-based  learning  environment. 
Restructuring  is  reflective.  It  involves  exploring  the  domain  in  depth,  forming  comparisons,  and 
integrating  across  related  domains.  Accretion  and  tuning  can  be  done  within  a  traditional 
training  setting.  Restructuring  is  a  professional  development  activity;  it  is  best  done  as  self- 
directed  study  or  as  part  of  formal  professional  military  education. 

Norman  also  comments  that  one  of  the  problems  associated  with  the  development  of  expertise  is 
making  trainees  want  to  do  the  hard  work  necessary  for  restructuring.  Ericsson  and  Chamess 
(1994)  support  Norman’s  view  and  assert  that  a  trainee’s  motivation  to  attend  to  the  task  and 
exert  effort  to  improve  his  or  her  performance  is  critical  to  development  as  an  expert  performer. 
Training  managers  delude  themselves  if  they  believe  that  skilled  performance  is  easy  and  can 
come  about  without  extended  effort  on  the  part  of  trainees. 


8 


The  bottom  line  for  the  previous  discussion  is  clear:  Practice  is  the  major  independent  variable 
in  skill  acquisition  and  the  development  of  expertise — but  not  just  any  kind  of  practice.  Ericsson 
and  Chamess  (1994)  assert  that  skilled  performance  is  not  an  automatic  consequence  of  simply 
having  more  experience  with  an  activity.  Skilled  perfonnance  is  developed  through  what  is 
referred  to  as  “deliberate  practice.”  Deliberate  practice  is  a  long  period  of  active  learning  during 
which  job  performers  refine  and  improve  their  skills  under  the  supervision  of  an  instructor  or 
coach.  It  requires  a  commitment  to  hours  and  hours  of  relevant  practice  with  expert  feedback. 

Several  times  throughout  the  previous  discussion,  we  have  alluded  to  the  importance  of  a  proper 
conceptual  framework  or  mental  model  in  learning  and  job  performance.  A  mental  model  is  a 
job  performer’s  internal  understanding  of  how  a  perfonnance  situation  works  or  fits  together.  It 
pennits  job  performers  to  make  sense  of  a  perfonnance  situation  and  his  or  her  role  in  successful 
performance.  Norman  (2002)  remarks  that  a  good  mental  model  permits  equipment  users  to 
predict  the  effects  of  their  actions.  Without  a  good  model,  users  perfonn  as  they  are  told  without 
really  knowing  why.  As  long  as  things  work,  they  can  manage.  However,  when  things  go 
wrong  or  when  the  unexpected  happens,  users  frequently  are  at  a  loss  as  to  how  to  proceed. 

Mental  models  also  can  be  an  aid  to  effective  instruction.  We  noted  above  that  a  proper  mental 
model  facilitates  accretion.  Without  one,  trainees  are  forced  to  engage  in  a  rote  memorization 
drill  that  is  tedious  and  error-prone.  Also,  material  learned  in  this  manner  is  not  easily  retained 
or  generalized.  Further,  in  the  tuning  phase  of  learning,  mental  models  serve  a  similar  purpose. 
They  help  trainees  make  sense  of  what  they  are  being  asked  to  do.  It  is  easier  to  learn  and  retain 
something  that  makes  sense  as  opposed  to  something  that  does  not.  A  word  of  caution  is  in  order 
here:  Trainees  will  develop  mental  models  on  their  own  if  not  provided  with  them  as  part  of  the 
instruction — and  these  may  not  be  accurate  or  entirely  appropriate.  Inappropriate  mental  models 
can  work  against  the  objectives  of  the  training  program. 

2.3  Training  Demons:  The  Enemies  of  Good  Training 

To  close  the  present  section,  we  provide  a  list  of  Training  Demons,  or  enemies  of  good  training. 
These  are  things  that  we  have  observed  routinely  in  AMD  and  Army  training.  They  illustrate 
what  can  happen  when  training  development  and  delivery  are  conducted  without  regard  for  the 
basics  as  described  in  the  previous  subsections. 

1 .  Inadequate  job  and  task  analysis:  Not  understanding  what  to  train.  A  competent  JTA  is  the 
primary  input  to  the  instructional  development  process.  Training  is  only  as  good  as  its 
analytical  basis.  To  the  extent  this  analysis  is  incomplete  or  inadequate,  trainers  run  the 
risks  associated  with  the  GIGO  phenomenon:  Garbage  In-Garbage  Out. 

2.  Misplaced  training  planning  emphases:  Media  dreams,  training  fads,  and  silver  bullets. 

We  remarked  earlier  that  exotic  media,  games,  simulations  and  other  “enhanced”  training 
environments  are  not  training  per  se.  They  can  be  used  to  support  learning,  but  their 
effective  use  is  predicated  on  a  comprehensive  understanding  of  (1)  what  must  be  trained, 
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(2)  a  proper  performance  setting  for  relevant  practice,  and  (3)  feedback  requirements.  In 
training,  there  are  no  silver  bullets. 

3.  Lock-step,  time-based  versus  criterion-referenced  instruction.  Much  of  AMD  and  Army 
training  is  lock-step  (everyone  moves  through  the  curriculum  at  the  same  pace  and  finishes 
at  the  same  time)  and  time-based  (shoe-horned  into  a  fixed  allotment  of  time).  Training-to- 
standard,  or  criterion-referenced  instruction,  does  not  lit  well  into  such  a  straight  jacket. 
People  do  not  learn  at  the  same  pace.  Course  lengths  must  be  based  on  a  consideration  of 
what  must  be  trained,  competency  requirements,  and  how  long  that  learning  process  is 
expected  to  take.  Arbitrarily  fixing  course  lengths  simply  means  that  some  knowledge  and 
skills  will  not  be  learned  to  standard,  and  this  will  likely  be  reflected  down-stream  in 
inadequate  on-the-job  performance.  Exit  criteria  from  training  must  be  some  level  of 
proficiency,  not  a  fixed  number  of  hours  of  class  time. 

4.  Content-based  training.  Content-based  training  is  most  often  encountered  in  academic 
instruction.  Its  purpose  is  to  teach  students  about  something,  not  to  teach  them  how  to  do 
something.  Classroom  training  is  not  an  adequate  approach  to  training  skills — how  to  do 
something.  People  learn  to  do  by  doing.  Content-based  training  is  often  a  workaround  for 
an  inadequate  JTA.  We  do  not  know  in  detail  what  the  job  requirements  will  be,  so  we 
teach  about  the  job  rather  than  how  to  do  it. 

5.  Not  understanding  that  free-play  is  not  deliberate  practice.  As  noted  earlier,  deliberate 
practice  is  essential  to  the  development  of  job  competence.  This  is  training  conducted 
under  the  close  supervision  of  a  master  performer  with  frequent  constructive  feedback. 
From  our  observation,  much  unit-based  training  in  AMD  is  more  like  free  play.  Trainees 
perform  within  the  context  of  a  scenario  without  close  supervision  and  the  relevant 
feedback  that  goes  with  it.  Based  on  the  JTA,  scenarios  must  be  written  to  provide  the  kind 
of  structured  training  and  practice  required  for  skill  development.  Further,  there  must  be 
multiple  scenarios  for  the  practice  of  any  particular  skill.  Hence,  scenario  construction, 
based  on  the  JTA,  is  a  high  priority  for  effective  training.  Free  play  affords  few 
opportunities  for  effective  learning  and  skill  development. 

6.  Inadequate  or  poorly-defined  standards.  A  review  of  AMD  and  Army  training  publications 
reveals  page  after  page  of  tasks,  conditions,  and  standards,  sometimes  very  detailed.  The 
task-standard  approach  to  assessing  training  success  might  be  appropriate  for  a  linear 
function  like  March  Order  and  Emplacement  or  System  Set-Up.  But,  for  a  complex, 
knowledge-based  job  like  AMD  ABO,  individual  task  performance  standards  are  not 
enough.  The  ABO  job  is  not  performed  as  a  series  of  individual,  sequential  tasks.  It  is 
entirely  possible  that  a  trainee  could  successfully  meet  each  task  perfonnance  standard  in 
sequence,  yet  be  “clueless”  about  overall  job  requirements — fighting  the  air  battle.  The 
OIF  BOI  alluded  to  this  possibility  with  its  remark  that  “The  system  [Patriot]  is  too  lethal 
to  be  placed  in  the  hands  of  crews  trained  to  such  a  limited  standard.”  In  a  knowledge- 
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based  setting  like  ABO,  standards  must  address  performance  requirements  for  the  whole 
job  or  for  meaningful  job  segments  as  opposed  to  individual  tasks.  Individual  task 
standards  might  be  useful  during  early  phases  of  ABO  training,  but  they  are  not  suitable  for 
later  tactical  training.  The  ABO  job  is  more  than  the  sum  of  its  individual  task  parts. 

7.  Lack  of  emphasis  on  the  AAR  process.  Proper  feedback  or  knowledge  of  results  is 
essential  to  learning.  With  respect  to  AMD  ABO,  research  indicates  that  proper  feedback 
is  a  critical  factor  in  improving  decision  accuracy  (Kozlowski,  1998).  In  a  discussion  of 
air-defense-specific  results,  Bisantz,  Kirlik,  Gay,  Phipps,  Walker,  and  Fisk  (1997)  assert 
that  knowledge-oriented  training  for  complex  judgment  tasks  must  address  the  skills 
required  to  consistently  execute  judgments  based  on  that  knowledge.  Trainees  must  be 
taught  why  their  decisions  were  good  or  bad.  This  requires  close  and  continuous  contact 
with  a  master  performer  acting  as  coach. 

8.  Training  folk  wisdom.  Folk  wisdom  refers  to  beliefs  about  a  subject  area  that  are  based  on 
personal  experience  uninformed  by  theory  or  broader  empirical  results.  In  the  training 
domain,  folk  wisdom  is  exhibited  in  statements  like  “Everybody’s  a  trainer”  or  “This  is  the 
way  we’ve  always  done  it.  It  worked  for  me,  and  it’ll  work  for  them.”  Roth  (1998) 
remarks  emphatically  that  job  performance  and  unit  experience  do  not  imply  instructional 
competence.  Training  and  learning  are  serious,  technical  activities.  Not  to  regard  them  as 
such  invites  many  of  the  problems  alluded  to  in  the  first  seven  of  these  Training  Demons. 


3.  AMD  Training  Dilemmas:  Issues  for  Research  and  Experimentation 


3,1  Overview 

If  the  AMD  ABO  job  setting  were  like  most  others  in  the  Army  or  like  those  that  dominated  in 
the  past,  it  would  be  possible  to  stop  our  training  discussion  here.  The  basics  described  in  the 
previous  section,  if  properly  applied,  would  be  sufficient  to  provide  competent  job  performers. 
However,  that  is  not  the  case.  Compared  with  routine  Army  jobs,  the  AMD  ABO  job  setting  has 
more  of  the  following  characteristics: 

1 .  Complex 

2.  Ambiguous  and  uncertain 

3.  Non-linear 

4.  Knowledge-intensive 
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Earlier,  we  commented  on  the  importance  of  a  comprehensive  and  up-to-date  JTA  in  developing 
good  training  for  a  job  like  AMD  ABO.  Two  of  the  characteristics  listed  above  make  conducting 
a  JTA  for  such  jobs  more  complex  than  with  routine  jobs.  First,  in  most  Army  jobs  (including 
many  AMD  jobs)  task  sequences  are  linear.  This  means  that  tasks  are  executed  in  a  designated 
sequence:  A  to  B  to  C  to  D,  and  so  on.  Branches  in  the  task  sequence  are  governed  by  relatively 
well-defined  rules:  If  (Situation),  Then  (Task  E),  Else  (Task  F).  Conventional  time  and  error 
metrics  can  be  used  as  the  basis  for  perfonnance  standards. 

In  a  non-linear  job,  the  task  sequence  is  not  so  well  defined.  Instead  of  linear  task  sequences, 
there  are  non-linear  task  networks.  Moreover,  for  knowledge-intensive  jobs,  conscious  problem 
solving  and  decision-making — based  on  cues  from  the  operating  environment — dictate  which 
actions  are  performed  and  in  what  order.  Non-linear,  knowledge-based  performances  also 
introduce  varying  degrees  of  equifinality  into  the  task  network.  Equifinality  means  that  there  is 
more  than  one  path  through  the  task  network  that  leads,  essentially,  to  the  same  outcome.  These 
aspects  of  the  AMD  ABO  job  do  not  fit  conveniently  into  traditional  JTA  concepts.  Moreover, 
traditional  task  time  and  error  metrics  can  be  problematic  in  the  presence  of  equifinality. 

The  job  characteristics  listed  above  also  require  that  training  for  the  ABO  job  be  approached 
differently  than  more  routine  activities.  A  failure  to  do  so  on  the  part  of  the  AMD  training 
community  was  the  principal  reason  for  the  OIF  BOI  criticism  that  Patriot  training  emphasized 
rote  drills  rather  than  flexible  problem  solving.  Jobs  characterized  by  the  features  listed  above 
cannot  be  adequately  trained  outside  of  a  problem-solving  context.  Such  jobs  also  place  a 
premium  on  jobholder  expertise.  Preparation  for  them  will  require  an  expansion  of  the  Branch’s 
views  regarding  training  in  the  direction  of  what  has  been  more  typically  regarded  as  education. 
Traditionally,  training  has  been  viewed  as  preparation  for  a  job  that  can  be  fully  characterized  by 
the  results  of  a  conventional  JTA.  Education,  on  the  other  hand,  has  been  viewed  as  appropriate 
for  jobs  that  cannot  be  so  neatly  circumscribed.  Under  a  supervisory  control  regimen,  operator- 
controller  jobs  have  evolved  away  from  traditional  operator  in  the  direction  of  system  manager. 
What  the  Army  has  traditionally  called  training  also  must  evolve  to  reflect  this  change  in  job 
orientation. 

Within  the  AMD  ABO  performance  domain,  we  have  identified  six  training-related  dilemmas  or 
issues  for  follow-on  research  and  experimentation.  These  are  issues  that  remain  unresolved  and 
thus  stand  in  the  way  of  moving  ABO  training  away  from  its  current  crew  drill  orientation  in  the 
direction  of  regarding  operator-controllers  as  managers  of  lethal  and  complex  systems.  These 
dilemmas  are: 

1 .  Initial  skill  development 

2.  Time  to  train  and  job  progression 

3.  Training  for  unreliable  automation 

4.  Team  training 
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5.  Adaptability 

6.  Mindfulness 

Each  of  these  issues  is  addressed  separately  in  the  subsections  to  follow. 

3.2  Initial  Skill  Development 

It  is  a  fact  of  life  that  future  AMD  C2  systems  will  involve  considerable  automation  and 
accompanying  HSC.  More  than  two  decades  ago,  Bainbridge  (1983)  first  commented  on  one  of 
the  apparent  ironies  of  automation:  An  automated  perfonnance  environment  often  does  not 
provide  a  good  training  setting.  From  a  skill  development  perspective,  automation  introduces  a 
special  problem  not  encountered  in  less  complex  systems.  In  manual  systems,  operators  can 
develop  their  skills  progressively  as  they  move  from  simple  operating  situations  to  more 
complex  tasks.  Automated  systems,  on  the  other  hand,  often  represent  an  all-or-nothing 
situation.  They  do  not  require — and  sometimes  prevent — direct  operator  participation  in  the 
control  process.  Therefore,  an  automated  performance  environment  might  not  support 
progressive  skill  development.  This  also  means  it  might  not  be  suitable  to  use  actual  equipment 
with  embedded  training  for  all  phases  of  operator  training. 

Klein  (2003)  discusses  a  second  skill  development  problem  associated  with  automated  systems. 
He  remarks  that  “smart”  technology  (like  that  associated  with  automation)  can  make  operator- 
controllers  “stupid”  (ineffective  as  decision  makers)  in  three  ways: 

1.  By  disabling  the  expertise  of  controllers  who  are  already  skilled. 

2.  By  slowing  the  rate  of  learning  so  users  do  not  develop  appropriate  levels  of  expertise. 

3.  By  reinforcing  dysfunctional  skills  that  will  interfere  with  users’  ability  to  achieve 
expertise  in  the  future. 

In  the  first  case,  Klein  notes  that  the  infonnation  screening  and  filtering  often  associated  with 
automation  prevent  operator-controllers  from  finding  the  information  they  need  to  make 
decisions.  Operator-controllers  thus  are  denied  access  to  some  or  all  of  the  decision-cueing 
information  essential  to  effective  decision  making.  One  might  say  that  operator-controller 
decision  making  can  be  inadvertently  short-circuited  through  data  display  and  access  decisions 
made  by  system  developers,  who  often  are  not  familiar  with  tactical  performance  requirements. 

Second,  the  same  information  screening  and  filtering  alluded  to  in  the  previous  paragraph  can 
deny  novice  or  journeyman  controllers  access  to  the  data  necessary  to  form  the  associations 
between  cue  sets  and  environmental  patterns  that  are  essential  to  effective  decision  making.  In 
the  first  case,  infonnation  filtering  prevents  current  experts  from  performing  as  experts.  That 
same  filtering  can  prevent  new  operator-controllers  from  ever  developing  into  experts. 

Klein’s  third  aspect  of  decision-making  impairment  involves  reinforcing  dysfunctional  coping 
practices  within  the  automated  environment.  Here,  operator-controllers  become  passive 


13 


recipients  of  information  from  the  machine.  Klein  notes  that  this  passivity  tends  to  make  new 
users  reluctant  to  work  around  problems  or  strike  out  on  their  own  to  become  true  experts.  They 
lack  the  background  to  do  so.  He  further  remarks  that  new  users  lose  or  never  develop  their 
ability  to  look  and  search  critically  within  the  tactical  enviromnent.  Smart  technology  has  made 
them  passive  and  ineffective  as  problem  solvers. 

Rasmussen  (1986)  remarks  that  in  an  automated  setting  the  skills  required  for  successful 
performance  during  extreme  situations  are  not  always  developed  or  maintained  during  nonnal 
training  or  operations.  He  cites  three  reasons  for  this  potential  problem: 

1 .  Unknown  but  latent  system  faults  cannot  be  simulated. 

2.  System  behavior  may  not  be  known  for  faults  that  can  be  predicted  but  have  not  been 
experienced. 

3.  What  was  abnonnal  becomes  routine. 

The  first  two  bullets  above  are  self-explanatory:  Things  will  happen  that  have  not  been 
anticipated  or  experienced  and  thus  cannot  be  explicitly  trained.  In  the  third  situation,  what  was 
once  considered  abnonnal  enters  the  training  program  and  becomes  routinized  over  time  and 
repetition.  The  tendency  here  is  not  to  regard  the  abnormal  situation  as  representative  of  a 
potential  class  of  abnonnal  situations  that  might  occur,  but  just  another  isolated  fault  that  has 
been  identified  and  conected.  This  illustrates  the  danger  of  simply  incorporating  unbriefed 
surprises  in  training  scenarios  without  emphasizing  these  anomalies  within  a  context  of 
adaptability  and  mindfulness. 

With  respect  to  initial  skill  development,  the  issue  to  be  resolved  going  forward  is  simply  stated 
as  follows:  Is  it  necessary  to  learn  first  in  a  less  automated  perfonnance  setting?  Can  jobholders 
adequately  master  their  ABO  role  if  training  is  conducted  solely  in  an  automated  practice 
environment?  The  research  results  in  this  area  suggest  that  automation  itself  might  be  an 
impediment  to  the  development  of  the  level  of  expertise  required  for  effective  ABO 
performance. 

3.3  Time  to  Train  and  Job  Progression 

Traditionally,  one  of  the  primary  articles  of  faith  in  Anny  training  is  that  personnel  must  be 
trained  effectively.  Trainees  must  acquire  all  of  the  skills  needed  to  perform  all  of  the  tasks 
needed  to  meet  the  missions  or  goals  and  objectives  of  the  system.  Not  to  do  so  is  to  open  the 
door  to  poor  on-the-job  performance  and  possibly  to  court  military  disaster.  In  section  2,  we 
noted  that  deliberate  practice  is  the  major  independent  variable  in  skill  acquisition.  Moreover, 
the  amount  of  time  devoted  to  training  must  be  based  on  a  consideration  of  what  must  be  trained 
and  standards  for  acceptable  on-the-job  performance.  Arbitrarily  fixing  school  course  lengths  or 
the  duration  of  on-the-job  training  (OJT)  means  that  some  knowledge  and  skills  will  not  be 
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learned  adequately,  and  this  deficiency  will  be  reflected  downstream  in  inadequate  on-the-job 
performance. 

Section  2  also  raises  another  controversial  topic:  The  amount  of  time  in  deliberate  practice 
required  to  progress  from  novice  to  introductory  master  performer.  A  figure  of  5,000  hours,  or 
two  years  of  full-time  instruction,  was  cited  as  the  minimum  amount  of  time  required  to  quality 
as  an  entry-level  expert.  Those  figures  being  on  the  table,  let  us  consider  how  much  training 
time  is  currently  allocated  for  U.S.  Army  Patriot  personnel  to  progress  from  novice  status  (no 
Patriot-specific  training)  to  qualification  as  a  Tactical  Control  Officer  (TCO)  or  Tactical  Control 
Assistant  (TCA).  Estimates  vary,  but  following  conversations  with  subject  matter  experts 
(SMEs),  we  arrived  at  the  following  times.  For  officers,  the  estimate  is  about  eight  months.  This 
includes  approximately  four  months  of  Patriot  Weapons  Track  Training  as  part  of  the  Officers’ 
Basic  Course  followed  by  as  little  as  four  months  of  OJT.  For  enlisted  personnel  (MOS  14E), 
the  Patriot  Fire  Control  Enhanced  Operator-Maintainer  course  during  Advanced  Individual 
Training  (AIT)  lasts  19  weeks.  However,  more  than  half  of  14E  AIT  is  taken  up  with 
maintenance  topics.  The  other  portion  is  taken  up  with  ABO  training.  Following  the  nearly  five 
months  of  AIT,  a  new  14E  could  qualify  as  a  TCA  with  as  little  as  four  months  of  OJT  at  his  or 
her  assigned  unit — nine  months  total.  These  estimates  are  clearly  short  of  the  two  years  cited  in 
the  literature  for  developing  in-depth  job  expertise. 

As  part  of  our  exploration  of  the  time-in-job-preparation  issue,  we  also  examined  training 
patterns  for  two  benchmark  cases: 

1 .  Israeli  Patriot 

2.  FAA  en  route  air  traffic  controllers  (ATCs) 

Israeli  Patriot  operators  first  undergo  four  months  of  initial  training,  which  qualifies  them  as  a 
low-level  TCA.  As  a  low-level  TCA,  trainees  are  allowed  to  observe  in  the  ECS  under 
supervision  and  perform  other  battery  duties.  Trainees  remain  low-level  TCAs  for  four  months 
to  one  year,  depending  on  their  perfonnance-based  skill  progression.  After  completing  their  stint 
as  a  low-level  TCA,  trainees  undergo  an  additional  two  months  of  high-level  TCA  training. 

Upon  successful  completion,  they  become  high-level  TCAs  and  perform  as  a  squad  leader. 

The  “best”  sergeants  to  complete  the  high-level  TCA  course  and  other  leadership  courses  are 
selected  as  officer  candidates.  Selection  as  an  officer  candidate  is  followed  by  a  three-month 
officer’s  basic  course  and  a  four-month  air  defense  (AD)  officer’s  course  (seven  months  total). 
Trainees  then  return  to  their  unit  to  serve  as  a  low-level  TCO.  As  a  low-level  TCO,  they  are  not 
assigned  to  an  active  crew  that  responds  to  real-world  alerts. 

After  an  additional  year  in  the  unit  with  extensive  simulation-based  training,  trainees  can  qualify 
for  promotion  to  first  lieutenant  (1LT).  Promotion  to  1LT  is  followed  by  two  additional  months 
of  training  in  AD  tactics.  Successful  completion  of  the  tactics  course  qualifies  the  trainees  as 
high-level  TCOs.  As  a  high-level  TCO,  the  officer  can  serve  during  alerts  as  part  of  a  three- 
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officer  ECS  crew  consisting  of  lLTs  and  captains.  In  total,  a  minimum  of  3 1  months  of  training 
interspersed  with  unit  experience  is  required  for  qualification  as  a  TCO. 

Additional  relevant  features  of  Israeli  Patriot  training  include  the  following: 

•  All  TCA  and  TCO  training  is  strictly  on  ABO 

•  ICC  Tactical  Directors  are  selected  from  tactical  ECS  crews  and  undergo  an  additional  six 
weeks  of  training 

•  OJT  is  simulation-based 

•  Periodic  hands-on  exams  are  required  following  certification 

•  Disqualification  from  alert  status  is  automatic  upon  failure  of  an  examination 

The  job  of  an  FAA  en  route  ATC  is  conceptually  similar  to  a  Patriot  operator-controller,  albeit 
with  different  ends  in  mind.  To  begin  the  process  of  becoming  an  en  route  controller,  job 
candidates  undergo  a  battery  of  pre-training  selection  instruments  looking  at  both  trainability  and 
aptitude  for  the  ATC  job.  Candidates  who  pass  the  pre -training  screening  phase  move  on  to  the 
FAA  Academy  in  Oklahoma  City,  OK,  for  four  months  of  intensive  resident  instruction  on  en 
route  operations.  Much  of  the  Academy’s  training  is  simulation-based  with  a  roughly  1 :2 
student-to-instructor  ratio.  One  instructor  serves  as  a  real-time  mentor,  while  the  other  performs 
discrete  skill  checks.  Essentially,  academy  hands-on  training  is  tutorial,  with  the  tutors  being 
retired  ATCs. 

Following  successful  completion  of  Academy  training,  controller  candidates  return  to  their 
assigned  center  for  an  additional  two  to  three  years  of  apprenticeship,  depending  on  the 
complexity  of  the  air  sectors  at  that  center.  Certification — being  able  to  sit  at  the  scope  and 
direct  traffic  alone  and  without  supervision — is  performance-based  and  also  depends  upon  a 
mentor’s  assessment  of  readiness  for  full  certification.  Total  time  to  full  perfonnance 
certification  as  an  en  route  ATC  is  28-40  months.  Should  a  controller  move  from  one  center  to 
another,  the  OJT  process  starts  over.  However,  once  qualified,  an  experienced  controller  may 
progress  through  the  OJT  sequence  faster  than  a  new  trainee. 

Job  preparation  times  in  each  of  these  cases  are  considerably  longer  than  the  time  allocated  for 
training  in  U.S.  Army  Patriot.  Both  benchmark  situations  also  are  consistent  with  the  literature 
in  that  more  than  two  years  of  preparation  precedes  being  declared  qualified  for  an  operational 
position.  The  benchmark  situations  also  illustrate  the  time-honored  preparation  sequence  for  a 
high-skill  job:  trainee  to  apprentice  to  journeyman,  with  extensive  job  sample  testing  along  the 
way.  Both  are  also  characterized  by  spending  a  considerable  amount  of  time  in  the  operational 
setting  apart  from  formal  training.  The  FAA  has  observed,  for  example,  that  spending  time  in 
the  job  setting  prior  to  the  onset  of  formal  training  has  a  positive  impact  on  perfonnance  during 
training.  Trainees  who  have  spent  time  in  the  organization  as  an  observer-apprentice  appear  to 
be  better  able  to  take  advantage  of  formal  training  when  it  is  offered.  They  progress  through  the 
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training  sequence  faster  and  academic  attrition  is  lower.  Spending  time  in  the  operational  setting 
apart  from  formal  training  permits  trainees  to  acquire  what  Sternberg  et  al.  (2000)  refer  to  as 
“tacit”  knowledge.  Tacit  knowledge  is  job-relevant  information  that  is  personally  important  to 
the  learner.  It  has  been  found  to  enhance  “practical  intelligence,”  or  the  ability  to  find  the  best  fit 
between  trainees  and  their  work  environment. 

We  are  not  saying  that  either  the  FAA  or  the  Israelis  have  the  “right”  answer  for  how  long 
training  should  take  for  a  job  like  that  of  Patriot  operator-controller.  It  is  worth  noting,  however, 
that  in  both  benchmark  instances  the  organizations  in  question  keep  their  job  candidates  in  a 
preparatory  status  far  longer  than  the  U.S.  Army — more  than  three  times  as  long.  Given  the 
negative  comments  on  Patriot  training  coming  out  of  OIF,  the  issue  of  training  time  and 
preparation  sequence  for  Patriot  operator-controllers  must  be  re-examined.  Training  time  is 
clearly  an  issue  for  research  and  experimentation.  It  must  not  be  dictated  solely  by  bureaucratic 
constraints  or  administrative  considerations. 

3.4  Training  for  Unreliable  Automation 

One  of  the  unmistakable  conclusions  from  OIF  combat  operations  is  that  the  Patriot  system’s 
automation  is  not  perfectly  reliable  (see  Hawley  &  Mares,  2006).  Reliability  is  defined  as  being 
dependable  or  capable  of  being  relied  upon.  Extending  this  basic  definition  to  real-time  C2  as  in 
AMD  battle  command,  a  reliable  automated  system  is  one  in  which  the  functions  assigned  to  the 
machine  are  performed  accurately  and  appropriately. 

Research  indicates  that  if  operator-controllers  have  a  choice,  trust  in  automation  detennines 
usage.  Simply  put,  operator-controllers  will  elect  not  to  use  a  system  they  do  not  trust  (Lee  & 
Moray,  1992).  This  same  body  of  research  also  suggests  that  trust  in  automation  is  affected  by 
the  same  factors  that  influence  trust  between  individuals:  effectiveness  and  reliability  (Muir, 
1988).  In  the  case  of  AMD  C2,  operator-controllers  do  not  have  a  choice.  They  must  use  the 
system  they  are  provided.  So  let  us  next  consider  the  issue  of  training  for  situations  in  which  the 
automation  is  not  perfectly  reliable. 

Rovira,  McGarry,  and  Parasuraman  (2002)  remark  that  one  of  the  true  “ironies”  of  automation  is 
that  the  more  reliable  the  automation,  the  greater  its  detrimental  effects  when  it  does  fail. 
Repeated  successful  use  lulls  operator-controllers  into  a  false  sense  of  security  or  complacency 
regarding  the  automation’s  perfonnance.  These  authors  also  report  results  indicating  that 
operators  become  over-reliant  on  automation  when  it  provides  decision  and  action  choices  for 
them  and  do  not  check  underlying  information  choices  as  carefully  as  when  such  choices  are  not 
provided  explicitly.  Near-perfect  automation  will  lead  to  a  situation  in  which  operator-controller 
decision  making  is  biased  in  favor  of  uncritically  accepting  the  automation’s  recommendations. 

Wickens,  Dixon,  and  Ambinder  (2005)  present  results  indicating  that  imperfect  automation  is 
manageable,  but  users  must  be  pre-warned  of  the  nature  and  source  of  the  automation’s 
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imperfections.  These  authors  also  caution  that  reliabilities  less  than  75%  are  worse  than  no 
automation  at  all,  and  can  provide  users  with  what  they  term  a  “concrete  life  preserver.” 

Cohen,  Parasuraman,  and  Freeman  (1997)  and  Masalonis  and  Parasuraman  (1999)  argue  that 
trust  in  automation  should  not  be  all-or-none,  but  graded  and  differentiated  according  to  the 
operational  context.  These  authors  refer  to  this  as  situation-specific  trust.  The  automation  may 
work  very  reliably  in  certain  contexts,  in  which  the  operator  should  use  it  and  trust  it.  But  in 
certain  other  cases,  that  the  operator-controller  has  been  trained  to  look  for,  the  automation’s 
recommendations  may  be  suspect.  Operators  should  be  told  to  assess  the  situation  and  take  the 
action  that  best  suits  the  context,  in  their  judgment.  If  operator-controllers  can  be  trained  to 
recognize  the  appropriate  context,  then  they  can  know  when  to  trust  the  automation  and  when  its 
recommendations  should  be  discounted. 

In  a  similar  vein,  Lee  and  See  (2004)  assert  that  automation  should  be  designed  for  appropriate 
as  opposed  to  greater  trust — based  on  the  automation’s  expected  reliability  in  handling  various 
functions.  These  authors  go  on  to  state  that  in  situations  involving  imperfect  automation, 
operator-controller  training  must  emphasize: 

1 .  Expected  system  reliability 

2.  The  mechanisms  underlying  potential  reliability  problems 

3.  How  usage  situations  interact  with  the  automation’s  technical  characteristics  to  affect 
reliability 

The  implications  of  this  brief  discussion  of  automation  reliability  and  its  impact  on  training  are 
clear:  System  developers  and  users  must  be  brutally  honest  regarding  automation  reliability. 
Extensive  tests  must  be  performed  to  detennine  those  situations  in  which  the  automation  does 
not  meet  design  criteria  for  reliability.  Boundaries  of  successful  system  perfonnance  must  be 
pushed.  Moreover,  the  mechanisms  underlying  unreliable  performance  must  also  be  explored. 
Commanders  and  operator-controllers  must  then  be  apprised  of  system  unreliability  patterns  and 
trained  in  situations  that  will  expose  them  to  system  imperfections.  Meeting  this  challenge  will 
be  a  tall  order.  It  is  not  clear  that  even  well  prepared  operator-controllers  will  be  able  to  make 
the  decisions  necessary  to  realize  situation-specific  trust.  Much  research  and  experimentation  on 
this  issue  is  required.  A  good  starting  point  for  this  work  might  be  the  Lens  Model  approach  to 
assessing  Dynamic  Decision  Making  performance  advocated  by  Bisantz  and  her  colleagues 
(Bisantz,  Kirlik,  Gay,  Phipps,  Walker,  &  Fisk,  2000;  Jha  &  Bisantz,  2001). 

3.5  Team  Training 

It  has  been  noted  that  units,  not  individuals  are  the  basis  of  warfare  competence  (DSB,  2003). 
This  is  not  to  say  that  individual  performance  is  not  important.  Individual  performance  is  an 
important  component  of  crew  and  unit  performance.  But  for  the  most  part,  individuals  do  not 
perform  missions.  Crews  and  units  do. 
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It  is  also  an  accepted  fact  that  a  crew  is  more  than  the  sum  of  its  individual  parts,  and  melding 
the  disparate  parts  into  a  functioning  unit  takes  time  working  together.  Proper  perfonnance  at 
the  crew  level  requires  that  individual  crewmembers  be  aware  of  their  interdependencies 
(Kozlowski,  1998).  A  variety  of  recent  research  and  operational  results  suggests  that  managing 
crew  interdependencies  may  require  team  process  training  similar  to  the  Crew  Resource 
Management  (CRM)  programs  prevalent  in  the  aviation  community.  Once  almost  exclusively  an 
aviation  program,  CRM  is  now  broadly  viewed  as  the  use  of  all  available  human,  informational, 
and  equipment  resources  toward  effective  and  efficient  operations  in  operational  domains 
dependant  on  crew  or  team  performance  (Helmreich,  Merritt,  &  Wilhelm,  1999).  In  the  aviation 
world,  CRM  skills  are  viewed  as  a  primary  line  of  defense  against  human  error  and  its 
consequences.  The  FA  A  also  uses  CRM  principles  to  foster  cooperation  and  coordination 
among  the  controllers  in  ATC  centers. 

Several  additional  points  regarding  the  development  of  effective  crews  are  listed  as  follows: 

1 .  Effective  team  training  is  based  on  solid  individual  technical  training 

2.  Time  to  train  is  a  key  neglected  issue  in  developing  effective  crews 

3.  Leaders  are  central  to  the  crew  development  process 

In  the  first  case,  it  goes  without  saying  that  a  team  is  no  more  able  than  its  least  effective 
member.  Hence,  competent  crews  first  require  competent  individuals.  Second,  as  we  noted  at 
the  outset,  developing  effective  crew  performance  takes  time  working  together.  It  is  essential 
that  training  plans  be  based  on  an  understanding  of  how  organizations  acquire  competence.  As 
shown  in  figure  2,  competence  is  developed  from  the  ground  up:  individuals,  then  crews,  and 
finally  multi-echelon  units.  Inadequate  training  at  any  point  in  the  performance  chain  can  short- 
circuit  the  process. 


19 


Unit 

Performance 


Desired  Unit 


Figure  2.  The  development  of  organizational  competence. 

Recent  research  also  emphasizes  the  importance  of  leaders  in  the  development  of  effective 
crews.  Kozlowski  (1998)  notes  that  leaders  foster  effective  crew  performance  by  (1)  melding 
individuals  into  a  coherent  team,  (2)  fostering  the  development  of  an  adaptive  network  of  roles, 
and  (3)  assisting  the  crew  in  becoming  a  self-learning  system.  These  three  features — coherency, 
an  adaptive  network  of  roles,  and  a  self-learning  system — differentiate  effective  teams  from  a 
simple  aggregation  of  individuals.  Effective  teams  work  together,  learn  as  a  group,  and  adjust 
their  intra-team  roles  to  reflect  changing  operational  circumstances. 

3.6  Training  for  Adaptability 

One  of  the  human  dimension  enablers  for  the  Army’s  future  force  is  individual  and  team 
adaptability.  The  primary  reason  for  wanting  adaptive  individuals  and  teams  is  to  cope  with  the 
uncertainty  that  is  expected  to  characterize  future  operations.  Individuals  and  teams  must  be  able 
to  make  the  necessary  modifications  to  meet  emergent  challenges.  In  the  AMD  domain,  crews 
must  expect  to  modify  or  replace  plans.  They  must  expect  to  improvise. 

Despite  the  recent  emphasis  on  adaptability,  little  has  been  written  about  what  it  means  to  be 
adaptive,  and  even  less  has  been  done  to  foster  the  development  of  adaptive  individuals  and 
teams  (Klein  &  Pierce,  2001).  These  authors  also  remark  that  much  of  what  takes  place  during 
routine  military  training  and  operations  has  an  effect  counter  to  what  is  desired.  That  is,  much  of 
current  training  and  operations  acts  to  produce  non- adaptive  individuals  and  crews.  Specifically, 
these  aspects  of  traditional  training  include: 

1 .  Training  for  mastery  of  task  routine  to  the  exclusion  of  problem  solving 

2.  Using  the  crawl,  walk,  run  method  of  training  while  avoiding  trainee  mistakes 
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In  the  first  case,  the  usual  emphasis  of  training  is  on  mastering  routines  rather  than  adaptive 
problem  solving.  Klein  and  Pierce  (2001)  refer  to  the  results  of  this  practice  as 
“experiosclerosis.”  Crews  believe  they  are  experts  and  combat  ready  because  they  are  good  at 
the  routines,  but  the  routines  can  prove  to  be  a  strait  jacket  during  combat.  Traditional  individual 
and  unit  evaluation  practices  reinforced  this  mistaken  belief  on  the  part  of  crews  and 
commanders  at  all  levels  by  focusing  only  on  satisfactory  performance  of  routine  drills.  OIF 
BOIs  specifically  criticized  Patriot  training  for  its  focus  on  crew  drills  while  not  addressing 
adaptive  problem  solving. 

The  crawl,  walk,  run  method  of  training  often  means  that  training  is  not  challenging  or 
threatening.  However,  trainees  learn  from  their  mistakes.  If  they  are  not  allowed  to  make  and 
learn  from  mistakes,  trainees  do  not  develop  the  mental  models  that  underlie  effective  problem 
solving  performance.  Klein  and  Pierce  note  that  learning  from  both  successes  and  failures  is 
rapid.  Accordingly,  advanced  scenarios  must  require  trainees  to  face  ambiguities  and  to 
reconsider  options  and  re-plan.  Crews  must  rehearse  familiar  scenarios  of  failure  and  strive  to 
imagine  novel  ones.  Failure,  in  present  usage,  means  situations  in  which  intended  results  were 
not  achieved  or  adverse  incidents  occurred.  Fratricides  like  those  that  occurred  during  OIF  are 
examples  of  adverse  incidents.  While  this  approach  might  sound  straightforward,  it  is  not  easily 
put  into  practice. 

Klein  and  Pierce  caution  that  while  most  crews  can  become  adaptive,  most  will  not  achieve  that 
level  of  expertise.  The  most  likely  reason  for  not  becoming  adaptive  has  to  do  with  the 
requirement  for  long  hours  of  deliberate  practice  and  reflection  discussed  in  section  2.  Aside 
from  trainee  motivation,  there  also  are  other  roadblocks  to  individuals  and  crews  becoming 
adaptive.  These  have  to  do  with  the  availability  of  master  performers  to  serve  as  mentors, 
training  equipment  availability,  adequate  training  scenarios,  and  a  variety  of  routine  bureaucratic 
constraints  and  administrative  practices.  It  can  be  argued  that  in  the  current  training 
environment,  adaptive  crews  are  likely  to  be  the  exception  rather  than  the  rule. 

3.7  Mindfulness 

The  challenge  facing  the  AMD  community  is  developing  units  that  are  capable  of  sustained  high 
reliability  in  a  complex,  unpredictable  operational  setting.  Daunting  as  this  challenge  might 
seem,  there  is  a  class  of  organizations  that  have  managed  to  do  just  that — maintain  high 
performance  in  complex  and  unpredictable  environments.  Weick  and  Sutcliffe  (2001)  refer  to 
such  organizations  as  high-reliability  organizations  (HROs).  Examples  of  HROs  include  air 
traffic  control  facilities,  nuclear  submarines,  and  aircraft  carrier  deck  operations.  As  one  FAA 
ATC  manager  put  it:  “99.9%  operational  reliability  is  not  good  enough  for  us.  That  would  mean 
we  would  have  several  dozen  ‘noticeable’  incidents  each  working  day.  And  that  would  bring  our 
operations  to  a  complete  halt”  (M.  Morrison,  personal  communication,  August,  30,  2005). 
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Weick  and  Sutcliffe  remark  that  HROs  are  characterized  by  ways  of  acting  and  leadership  styles 
that  enable  them  to  manage  the  unexpected  better  than  most  kinds  of  organizations.  These 
authors  note  that  HROs  manage  the  unexpected  through  five  processes: 

1 .  Preoccupation  with  failures  rather  than  successes.  Perfonnance  lapses  and  near  misses  are 
treated  as  symptoms  that  something  is  wrong  with  the  system — something  that  could  have 
severe  consequences  if  a  combination  of  lapses  were  to  coincide  at  the  same  point  in  time. 
Consequently,  HROs  study  incidents  and  near  misses  for  what  can  be  learned.  They  are 
wary  of  the  potentially  disastrous  consequences  of  complacency  and  a  drift  into  automatic 
processing. 

2.  Reluctance  to  simplify  interpretations.  HROs  are  reluctant  to  accept  simplified 
explanations  of  potential  trouble  spots.  They  encourage  critical  inquiry  across  the 
organization,  and  are  skeptical  of  “received  wisdom”  from  outside  sources  who  are  often 
less  knowledgeable  about  line  conditions. 

3.  Sensitivity  to  operations.  HROs  are  characterized  by  sensitivity  to  line  operations.  This  is 
defined  as  a  focus  on  understanding  the  situational  aspects  of  organizations  operations. 
They  do  not  permit  a  disconnect  between  operations  as  viewed  from  the  top  and  operations 
as  implemented  on  the  front  line. 

4.  Commitment  to  resilience.  HROs  accept  that  no  system  is  perfect.  They  expect  that  errors 
will  occur  and  train  their  crews  to  recognize  and  recover  from  them.  Following  adverse 
incidents  or  near  misses,  HROs  focus  on  system-wide  fixes  rather  than  item-by-item 
repairs.  This  requires  a  deep  knowledge  of  the  technology,  the  system,  crews,  and 
individuals.  Consequently,  HROs  place  a  premium  on  expertise:  personnel  with  extensive 
experience,  recombination  skills,  and  training. 

5.  Deference  to  expertise,  as  exhibited  by  encouragement  of  a  fluid  decision-making  system. 
HROs  recognize  that  rigid  hierarchies  are  vulnerable  to  error.  Rigid  C2  structures  are 
suitable  for  a  stable  world,  but  can  be  an  impediment  when  adaptability  is  required.  HROs, 
on  the  other  hand,  push  decision  making  down  and  around  in  the  organization.  Decisions 
are  made  on  the  front  line;  authority  migrates  to  the  people  with  the  most  expertise, 
regardless  of  rank. 

Together,  these  five  processes  produce  an  attitude  of  mindfulness  or  “intelligent  wariness.”  To 
be  mindful  is  to  have  an  awareness  of  detail  and  an  enhanced  ability  to  identify  and  prevent 
errors  that  could  escalate  into  an  adverse  event.  How  can  an  organization  manage  the 
unexpected?  Weick  and  Sutcliffe  assert  that  organizations  can  manage  the  unexpected  by  acting 
more  like  a  HRO. 

Desirable  as  it  might  be,  acting  more  like  a  HRO  is  neither  simple  nor  easy.  For  example,  Weick 
and  Sutcliffe  caution  that  it  is  difficult  for  individuals  and  crews  to  remain  chronically  wary 
about  their  operations.  Moreover,  several  of  the  HROs  noted  in  the  previous  discussion  (e.g., 
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nuclear  submarines  and  air  traffic  control)  have  had  to  create  distinct  supporting  subcultures  that 
sometimes  put  them  at  odds  with  their  parent  organizations  (see  Bierly,  1995).  They  have  had  to 
take  and  hold  to  the  position  that  effective  and  reliable  line  operations  come  first  and  must  be 
emphasized  above  broader  organizational  policies  and  practices.  Their  imperative  is  to  make 
their  operations  as  effective  and  robust  as  practical  in  the  face  of  sometimes  unpredictable 
engineering,  human,  and  operational  vulnerabilities. 


4.  A  Path  Forward  for  AMD 


4.1  Overview 

The  final  section  of  the  report  concerns  a  path  forward  for  AMD.  In  developing  this  section,  it  is 
not  our  intent  to  tell  others  their  business.  That  is,  it  is  not  our  intent  to  get  deeply  into  Anny  or 
AMD  training  design  and  development  processes,  other  than  to  suggest  areas  where 
improvements  to  increase  effectiveness  and  organizational  reliability  are  indicated.  These 
suggestions  for  improvement  are  taken  from  the  human  perfonnance  and  training  literature  as 
well  as  from  lessons  learned  in  parallel  organizations.  A  parallel  organization  is  one  engaged  in 
operations  and  training  for  jobs  similar  to  AMD  operator-controllers.  Again,  the  focus  of  the 
following  discussion  is  training  for  ABO. 

The  discussion  to  follow  is  also  presented  in  two  parts:  (1)  training  for  routine  operations  and 
(2)  training  for  adaptability.  Training  for  routine  operations  addresses  enhancements  directed  at 
improving  effectiveness  and  reliability  in  situations  involving  little  ambiguity  or  uncertainty. 

The  intent  here  is  to  develop  personnel  and  crews  well  practiced  in  the  basics  of  AMD 
operations.  Training  in  advanced  aspects  of  AMD  operations  such  as  coping  with  unreliable 
automation  must  be  based  on  a  firm  foundation  in  AMD  and  system  fundamentals. 

Training  for  adaptability  is  based  upon  competent  baseline  performance  by  individuals  and 
crews.  It  might  be  tenned  advanced  training  for  situations  that  are  non-routine  or  out-of-the- 
ordinary.  As  the  name  implies,  it  is  intended  to  foster  the  adaptability  and  mindfulness  necessary 
to  cope  with  unreliable  automation  and  the  inevitable  surprises  of  combat  operations. 

4.2  Training  for  Routine  Expertise 

Our  observations  of  AMD  training  and  operations  plus  material  from  the  literature  cited  in  the 
previous  sections  suggest  six  areas  for  improvement  in  AMD  training  for  routine  operations. 
These  points  are  listed  and  discussed  as  follows. 

1 .  Job  and  task  analysis.  We  remarked  earlier  that  training  design  can  be  no  better  than  the 
JTA  on  which  it  is  based.  Training  developers  must  understand  the  performance  context 
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and  what  trainees  must  learn  to  perform  competently  in  that  environment.  This  goal  cannot 
be  met  if  the  JTAs  upon  which  training  are  based  are  simply  updates  of  previous  JTA 
material.  The  first  and  second  reports  in  this  series  (Hawley,  Mares,  &  Giammanco,  2005; 
Hawley  &  Mares,  2006)  emphasize  that  automated  operations  present  a  different 
performance  setting  from  that  associated  with  traditional  manual  control.  These 
differences  must  be  reflected  in  training  and  job  aiding.  To  the  extent  these  performance 
changes  are  not  addressed  during  the  JTA  and  subsequent  training  development,  the 
resulting  training  will  be  inappropriate  and  sometimes  ineffective  vis-a-vis  actual 
performance  requirements. 

2.  Scenario  content.  We  noted  previously  that  in  training  for  a  job  like  AMD  operator- 
controller,  the  scenario  is  the  curriculum.  This  means  that  training  scenarios  must  be 
carefully  scripted  to  cue  the  decisions  and  performances  indicated  in  the  JTA  as  being 
critical.  Training  managers  must  also  be  aware  that  training  scenarios  have  a  limited  shelf 
life.  After  more  than  about  three  uses  with  the  same  group  of  trainees,  they  lose  their 
training  value.  Trainees  recognize  the  scenario  and  begin  to  “game”  it.  Gaming  scenarios 
destroys  their  learning  value,  since  gaming  rather  than  learning  becomes  the  objective  of 
the  exercise. 

3.  Performance  standards.  Training  managers  must  attend  to  the  development  of  operational 
performance  standards  for  scenarios  and  for  success  at  the  various  certification  levels. 

Skill  checks  for  individual  tasks  or  portions  of  tasks  are  important  in  skill  development,  but 
they  are  not  sufficient  for  judging  job  competence.  Standards  must  be  objectively  stated  in 
tenns  of  successful  performance  against  benchmark  scenarios  having  stated  performance 
objectives  and  known  difficulty  levels.  In  ABO  training,  the  whole  defined  as  competent 
job  perfonnance  is  more  than  the  sum  of  its  individual  task  parts.  Standards  must  also  be 
applied  uniformly  in  institutional  and  unit  training.  Trainers  and  commanders  must  know 
where  the  bar  defining  competent  job  performance  is  set. 

4.  Time  to  reach  desired  competency  levels.  Time  to  train  to  desired  competency  levels  is 
one  of  recurring  problems  in  Army  and  military  training  in  general  (DSB,  2003).  Training 
times  must  be  detennined  on  the  basis  of  how  long  individuals  take  to  reach  desired 
competency  levels.  A  situation  must  not  be  allowed  to  exist  in  which  training  is 
shoehorned  into  available  times  or  is  fixed  arbitrarily  on  the  basis  of  bureaucratic  or 
administrative  considerations.  If  such  considerations  are  pennitted  to  dictate  training 
times,  the  result  will  be  incomplete  training  and  inadequate  on-the-job  performance.  In 
training,  there  is  no  way  to  escape  the  fact  that  you  get  what  you  pay  for. 

5.  Qualified  instructors.  There  is  an  old  adage  that  it  is  hard  for  an  apprentice  to  learn  if  there 
are  no  masters.  Personnel  assigned  to  training  jobs  in  the  institution  and  in  units  must  be 
expert  job  performers  (EJPs).  An  EJP  is  a  SME  who  also  has  recent  and  relevant  on-the- 
job  experience.  Their  job  is  to  teach  trainees  how  to  do  the  job,  not  just  to  teach  them 
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about  the  job.  Consideration  must  also  be  given  to  the  teaching  ability  of  potential 
instructors.  Roth  (1998)  cautions,  for  example,  that  system  experience  is  not  a  guarantor  of 
effective  instructional  capability.  Also,  Nonnan  (1993)  cites  results  indicating  that  a 
course’s  reputation  is  a  major  determinant  of  trainee  motivation.  If  a  course  has  a  good 
reputation  that  word  will  spread  and  trainees  will  come  to  the  course  motivated  and 
prepared  to  leam.  On  the  other  hand,  if  a  course  has  a  bad  reputation,  that  fact  will  be 
reflected  in  an  unwillingness  to  come  or  attend  to  the  material.  Trainees  will  use  any 
excuse  to  get  out  of  attending.  Requiring  attendance  will  not  always  result  in  effective 
learning. 

6.  The  AAR  process.  A  generally  accepted  axiom  of  experimental  psychology  is  that  no 
learning  takes  place  in  the  absence  of  feedback.  This  simple  fact  underlies  much  of  the 
success  attributed  to  the  Army’s  training  at  Combat  Training  Centers  (CTCs)  such  as  the 
National  Training  Center  (NTC).  At  the  CTCs,  considerable  effort  goes  into  the  post¬ 
exercise  AAR.  Training  design  can  be  exceptional;  scenarios  can  be  great;  but  if  the  AAR 
process  is  deficient,  learning  will  be  inadequate.  The  same  relationship  applies  to  lower 
echelon  training  conducted  in  institutions  and  units  outside  the  CTC  context.  Good  AARs 
go  well  behind  a  mere  replay  or  recap  of  what  went  on.  They  consist  of  a  penetrating 
behavioral  critique  that  is  designed  to  change  the  way  trainees  think  about  the 
performances  in  question.  Trainees  must  be  told  not  only  that  a  particular  performance  was 
right  or  wrong,  but  also  why  that  performance  was  right  or  wrong.  Good  AARs  are  critical 
to  tuning  the  mental  models  upon  which  future  performance  will  be  based. 

4.3  Training  for  Adaptive  Expertise 

As  noted  previously,  training  for  adaptability  is  intended  to  foster  the  ability  to  think  outside  the 
box  defined  by  routine  crew  drills.  It  is  intended  to  provide  operator-controllers  with  the  ability 
to  cope  with  unreliable  automation  and  the  inevitable  unforeseen  events  that  characterize  combat 
operations.  In  one  sense,  the  mechanisms  underlying  training  for  adaptability  are 
straightforward.  As  the  DSB  recommended,  trainees  should  face  confusing  and  complex 
scenarios  that  contain  unbriefed  surprises.  Such  scenarios  must  push  trainees  and  crews  outside 
their  comfort  zone  and  stress  problem  solving  over  routine  operational  processes.  Crews 
working  together  must  recognize  when  a  situation  is  not  ordinary  and  requires  problem-solving 
intervention.  If  there  is  a  key  to  achieving  this  result,  the  operative  word  is  expertise.  Not  to 
confront  issues  associated  with  expertise  as  defined  in  this  report  invites  a  drift  toward  automatic 
processing  and  the  kinds  of  performance  problems  that  led  to  the  Patriot  Vigilance  effort. 

As  desirable  and  straightforward  as  training  for  adaptive  expertise  might  seem,  producing 
adaptive  individuals  and  crews  will  not  be  a  simple  undertaking.  We  cite  again  Klein  and 
Pierce’s  (2001)  caution  that  most  crews  can  become  adaptive,  but  most  will  not.  Why  not?  In 
our  observation,  three  roadblocks  stand  in  the  way: 
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1 .  Time  and  job  progression  practices 

2.  Training  quality 

3.  Trainee  motivation 

First,  achieving  adaptive  expertise  will  require  more  time  for  training  than  the  Anny  has 
traditionally  allocated  for  AMD  operator-controller  job  preparation.  Simply  put,  there  is  no  way 
to  avoid  the  5,000  hour  rule  that  applies  in  other  high-skill  situations.  Further,  intra-unit  job 
progression  patterns  will  have  to  change.  Operator-controller  trainees  will  have  to  spend  an 
extended  period  in  an  apprentice  status  while  they  acquire  the  skills  and  experience  necessary  for 
effective  job  perfonnance.  Unit  metrics  regarding  qualified  crews  will  have  to  change,  and  it  is 
not  certain  that  current  personnel  management  practices  and  concerns  can  accommodate  such  a 
requirement. 

Second,  many  qualitative  aspects  of  operator-controller  training  will  have  to  change.  To  begin, 
training  for  what  we  have  termed  routine  operations  will  have  to  be  more  rigorous  and 
performance-oriented  than  at  present.  Training  content  and  scenarios  must  reflect  job 
requirements,  and  standards  must  be  rigorously  applied  across  the  board.  Introductory,  baseline 
training  will  have  to  be  followed  by  crew-oriented  training  that  emphasizes  fluid  decision¬ 
making  within  an  adaptive  network  of  roles.  This  will  require  intact  crews,  special  training  such 
as  CRM,  competent  instructors,  and — above  all — time  to  form  this  collective  expertise.  Crew 
leadership  also  is  important  to  adaptive  team  performance. 

Third,  trainees  must  be  motivated  to  develop  the  deep  expertise  in  AMD  technology,  weapons 
systems,  and  operations  necessary  to  inform  the  decision  processes  that  characterize  being 
adaptive.  This  is  the  reflective  aspect  of  job  preparation.  As  we  noted  in  the  previous  section, 
the  proper  setting  for  such  preparation  is  self-directed  study  and  professional  military  education. 
Trainees,  commanders,  and  the  general  Anny  culture  must  accept  that  preparation  for  a  job  like 
operator-controller  involves  no  less  professionalism  than  preparation  for  command  or  any  other 
high-skill  job.  The  topics  and  focus  are  different,  but  the  preparatory  requirements  are  similar. 
We  have  discussed  them  extensively  in  this  report. 

An  example  from  the  Army  Aviation  community  might  clarify  and  illustrate  this  third  point. 
Army  Aviation  uses  a  special  Warrant  Officer  career  progression  track  to  increase 
professionalism  and  expertise  among  enlisted  aviators.  Warrant  Officer  aviators  are  officers,  are 
highly  selected,  are  granted  honor  and  deference  by  all  concerned,  and  are  expected  to  be  highly 
trained  professionals  who  operate  at  the  highest  levels  of  expertise.  Moreover,  it  is  anecdotally 
reported  that  instructor  pilots  demand  more  of  Warrant  Officer  students  than  of  Commissioned 
Officer  students  (D.  M.  Johnson,  personal  communication,  January  10,  2006).  Commissioned 
Officer  students  eventually  grow  up  and  become  staff  officers,  administrators,  and  commanders; 
but  Warrant  Officer  students  develop  as  expert  aviators  and  remain  in  this  role  throughout  their 
career.  The  emerging  job  of  an  AMD  operator-controller  is  no  less  demanding  than  that  of  an 
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Army  Aviator,  and  skill  and  professional  development  for  these  personnel  must  be  approached  in 
a  similar  manner. 

4.4  The  Training  Challenge 

Achieving  the  objectives  cited  in  the  previous  sections  will  not  be  easy.  To  begin,  commanders 
and  training  managers  must  recognize  that  the  Army’s  crew  drill  mentality  is  a  major  part  of  the 
problem  associated  with  preparing  Soldiers  for  knowledge-intensive  jobs.  As  Klein  and  Pierce 
(2001)  caution,  the  crew  drill  mentality  discourages  adaptive  problems  solving  and  almost 
guarantees  a  drift  toward  automatic,  unthinking  processing.  Its  mantra  is,  “This  is  the  way  we  do 
it,  and  any  other  way  is  wrong.”  Crew  drills  are  appropriate  for  some  job  situations,  but 
inappropriate  in  others.  They  are  not  suitable  for  ABO  training. 

In  the  report  Training  for  Future  Conflicts,  the  DSB  asserts  that  the  future  will  require  that  more 
of  our  people  do  new  and  more  complicated  things  (DSB,  2003).  That  same  report  also  remarks 
that  meeting  this  challenge  amounts  to  a  “qualitative  change  in  the  demands  upon  our  people  that 
cannot  be  supported  by  traditional  training  practices”  (p.  6).  The  DSB  report  concluded  that 
training  transformation  to  support  warfare  transformation  will  be  a  challenging  undertaking. 

Our  assessment  of  the  path  forward  for  AMD  training  is  consistent  with  the  DSB’s  conclusion. 
Technology  and  the  operating  environment  are  driving  changes  in  weapons  systems,  and  these 
combined  changes  require  a  change  in  training  concepts  and  practices.  Not  to  change  means  that 
AMD  training  will  become  increasingly  unsuited  to  operational  requirements.  The  2003  DSB 
report  cautions  that  training  failures  can  negate  promising  hardware  and  technologies.  That 
warning  applies  directly  to  the  technology-intensive  AMD  situation. 
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